TADA Module 1: Training for Intermediate/Advanced R Users
TADA Team
2024-06-03
Source:vignettes/TADAModule1_AdvancedTraining.Rmd
TADAModule1_AdvancedTraining.RmdWelcome!
Thank you for your interest in Tools for Automated Data Analysis (TADA). TADA is an open-source tool set built in the R programming language and available for anyone to download and edit to their specific needs. This TADA Module 1: Training for Intermediate/Advanced R Users RMarkdown document (learn more about RMarkdown) walks through how to download the TADA R package from GitHub, access and parameterize several important functions with a sample data frame, and create basic visualizations. The workflow is similar to a funnel: at each decision point, data that fail QC checks are removed from the core data frame and placed in a separate data frame, while data that pass are carried into the next step. At the end of the QC checks, the user should be confident that their data are properly documented and applicable to the analysis at hand.
Note: TADA is still under development. New functionality is added weekly, and sometimes we need to make bug fixes in response to tester and user feedback. We appreciate your feedback, patience, and interest in these helpful tools.
Customize or contribute
TADA is housed in a repository on GitHub. Users desiring to review the base code and customize the package for their own purposes may:
Clone the repository using Git
Open the repository using GitHub Desktop, or
Download a zip file of the repository to their desktop.
Interested in contributing to the TADA package? The TADA team highly encourages input and development from users. Check out the Contributing page on the TADA GitHub site for guidance on collaboration conventions.
Install and setup
Users can install the TADA package from GitHub into their R library
using the remotes package. Copy and paste the code below
into your R or RStudio console to download and install.
install.packages("remotes",
repos = "http://cran.us.r-project.org"
)
library(remotes)TADA package relies on other packages, therefore you may be prompted in the console to update dependency packages that have more recent versions available. If you see this prompt, it is recommended to update all of them (enter 1 into the console).
remotes::install_github("USEPA/TADA",
ref = "develop",
dependencies = TRUE
)It’s that easy! The most stable branch for TADA right now is the
develop branch. Contributors generally create their own branches based
on develop, make some improvements, and then submit a pull request to be
reviewed by the TADA Team. Once approved, updates are then merged into
the develop branch. However, you are welcome to download any branch
you’d like using the ref input in
install_github (see code chunk above). This functionality
is mainly only useful to TADA package developers/contributors.
The following code block ensures the additional packages needed to
run the code in this RMarkdown document are loaded. However, users may
also use the package name:: package function notation to
avoid the list of library() calls.
list.of.packages <- c("tidyverse")
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[, "Package"])]
if (length(new.packages)) install.packages(new.packages)
library(tidyverse)Help pages
All TADA R package functions have their own individual help pages,
listed on the Function
reference page on the GitHub site. Users can also access the help
page for a given function in R or RStudio using the following format
(example below): ?TADA::[name of TADA function]
?TADA::TADA_DataRetrievalUpload data
Now let’s start using the TADA R package functions. The first step is to bring a data frame into the R environment. TADA is designed to work with Water Quality Portal (WQP) data. This means that all of its functions will look for WQP column names and create new TADA-specific columns based on these elements. Users may upload their own custom data frame into R for use with TADA by ensuring their column names and data formats (e.g. numeric, character) align with WQP profiles.
If you are interested in reviewing the column headers and formats
required to run TADA, use the function below, which saves an example
spreadsheet to the user’s working directory. You can also take a look at
an example data frame, like TADA::Data_Nutrients_UT to get
an idea of the data structure and format.
getwd() # find your working directory## [1] "/home/runner/work/TADA/TADA/vignettes"
template <- TADA::TADA_GetTemplate() # download template to working directory
# uncomment below to review example data frame
# Data_Nutrients_UT <- TADA::Data_Nutrients_UTTADA::TADA_DataRetrieval is built upon USGS’s
dataRetrieval::readWQPdata function within the
dataRetrieval package, which uses web service calls to bring WQP data
into the R environment. Additionally,
TADA::TADA_DataRetrieval performs some basic quality
control checks via TADA::TADA_AutoClean on the data using
new TADA-specific columns to preserve the original data frame:
Converts key character columns to ALL CAPS for easier harmonization and validation.
Identifies different classes of result values (numeric, text, percentage, comma-separated numeric, greater than/less than, numbers preceded by a tilde, etc.) and converts values to numeric where feasible.
Unifies result and depth units to common units to improve ease of data harmonization. See
?TADA::TADA_ConvertResultUnitsand?TADA::TADA_ConvertDepthUnitsfor more information on these processes. These functions can also be run separately if the user wishes to convert result or depth values to different units.
Let’s give it a try. Setting applyautoclean to TRUE in
TADA:TADA_DataRetrieval means that the basic quality
control steps described above are run. See
?TADA::TADA_AutoClean for more details.
TADA::TADA_DataRetrieval follows similar parameterization
to the dataRetrieval package function
dataRetrieval::readWQPdata, but check out the help
page or enter ?TADA::TADA_DataRetrieval into the
console for more information about input parameters and to see several
examples.
# download example data
# dataset_0 <- TADA::TADA_DataRetrieval(
# organization = c("REDLAKE_WQX", "SFNOES_WQX", "PUEBLO_POJOAQUE", "FONDULAC_WQX", "PUEBLOOFTESUQUE", "CNENVSER"),
# startDate = "2018-01-01",
# endDate = "2023-01-01")
# For brevity, we'll skip pinging the WQP and instead load the example data frame:
dataset_0 <- TADA::Data_6Tribes_5y
# Let's take a look at all of the TADA-created columns:
names(dataset_0)[grepl("TADA.", names(dataset_0))]## [1] "TADA.ActivityMediaName"
## [2] "TADA.ResultSampleFractionText"
## [3] "TADA.CharacteristicName"
## [4] "TADA.MethodSpeciationName"
## [5] "TADA.ComparableDataIdentifier"
## [6] "TADA.ResultMeasureValue"
## [7] "TADA.ResultMeasureValueDataTypes.Flag"
## [8] "TADA.ResultMeasure.MeasureUnitCode"
## [9] "TADA.WQXResultUnitConversion"
## [10] "TADA.DetectionQuantitationLimitMeasure.MeasureValue"
## [11] "TADA.DetectionQuantitationLimitMeasure.MeasureValueDataTypes.Flag"
## [12] "TADA.DetectionQuantitationLimitMeasure.MeasureUnitCode"
## [13] "TADA.ResultDepthHeightMeasure.MeasureValue"
## [14] "TADA.ResultDepthHeightMeasure.MeasureValueDataTypes.Flag"
## [15] "TADA.ResultDepthHeightMeasure.MeasureUnitCode"
## [16] "TADA.ActivityDepthHeightMeasure.MeasureValue"
## [17] "TADA.ActivityDepthHeightMeasure.MeasureValueDataTypes.Flag"
## [18] "TADA.ActivityDepthHeightMeasure.MeasureUnitCode"
## [19] "TADA.ActivityTopDepthHeightMeasure.MeasureValue"
## [20] "TADA.ActivityTopDepthHeightMeasure.MeasureValueDataTypes.Flag"
## [21] "TADA.ActivityTopDepthHeightMeasure.MeasureUnitCode"
## [22] "TADA.ActivityBottomDepthHeightMeasure.MeasureValue"
## [23] "TADA.ActivityBottomDepthHeightMeasure.MeasureValueDataTypes.Flag"
## [24] "TADA.ActivityBottomDepthHeightMeasure.MeasureUnitCode"
## [25] "TADA.LatitudeMeasure"
## [26] "TADA.LongitudeMeasure"Currently, the TADA::TADA_DataRetrieval function
combines three WQP data profiles: Sample Results (Physical/Chemical),
Site data, and Project data. This ensures that all important quality
control columns are included in the data frame.
Note: USGS and EPA are working together to create
WQP 3.0 data profiles. Once released, one data profile will contain the
columns critical to TADA, removing the need to combine profiles in this
first step. TADA package users likely will not notice a difference in
their usage of the TADA::TADA_DataRetrieval function, but
it will simplify the steps needed to upload a custom or WQP
GUI-downloaded data frame into the R package.
Initial data review
Now that we’ve pulled the data into the R session, let’s take a look
at it. Note that any column names with the “TADA.” prefix were generated
from the TADA::TADA_DataRetrieval function.
First, always good to take a look at the data frame dimensions.
Question 1: What are the dimensions of your data frame?
dim(dataset_0) # returns x and of x (as the numbers of rows and columns respectively)## [1] 134050 149Before we start filtering and flagging our data, let’s create a
function (dimCheck) that performs dimension checks between
the results that pass each filter or QC flag check (and are retained)
and those that do not (and are removed). These dimension checks ensure
that the total number of rows in the original input data frame
(all_result_num) equal the the total number of rows added
up between the passing (pass_data) and removed
(fail_data) data frames.
# defining a dimension check function that compares removed and retained data dimensions against the initial data input
dimCheck <- function(all_result_num, pass_data, fail_data, checkName) {
# check result numbers after split
final_result_num <- dim(pass_data)[1] + dim(fail_data)[1]
# always good to do a dimension check
if (!all_result_num == final_result_num) {
print(paste0("Help! Results do not add up between data frame and removed after ", checkName, " check."))
} else {
print(paste0("Good to go. Zero results created or destroyed in ", checkName, " check."))
}
}
# let's first get the total number of rows in the data frame.
all_result_num <- dim(dataset_0)[1]Next, we can use the TADA::TADA_FieldCounts() function
to see how many unique values are contained within each column of the
data frame. The function can either return all column counts, most, or
just the key columns. We’ll try the input with
display = "key" and display = "all". Enter
?TADA::TADA_FieldCounts() into the console for more
information on this function.
Question 2: Which column should have a unique value in every row and why?
key_counts <- TADA::TADA_FieldCounts(dataset_0, display = "key")
key_counts## Fields Count
## 1 ActivityCommentText 805
## 2 TADA.CharacteristicName 143
## 3 ProjectName 30
## 4 LaboratoryName 9
## 5 DetectionQuantitationLimitTypeName 7
## 6 ActivityTypeCode 6
## 7 OrganizationFormalName 6
## 8 MonitoringLocationTypeName 5
## 9 StateCode 4
## 10 TADA.ActivityMediaName 3
## 11 ActivityMediaSubdivisionName 3
all_counts <- TADA::TADA_FieldCounts(dataset_0, display = "all")
all_counts## Fields Count
## 1 ResultIdentifier 134050
## 2 TADA.ResultMeasureValue 38999
## 3 ResultMeasureValue 35609
## 4 ActivityIdentifier 19025
## 5 ResultDetectionQuantitationLimitUrl 12311
## 6 ActivityStartDateTime 11922
## 7 LastUpdated 6665
## 8 ActivityStartTime.Time 3737
## 9 TADA.ResultDepthHeightMeasure.MeasureValue 3608
## 10 ResultDepthHeightMeasure.MeasureValue 3519
## 11 ActivityEndDateTime 1026
## 12 ActivityEndTime.Time 1001
## 13 ActivityCommentText 805
## 14 ActivityStartDate 756
## 15 AnalysisStartDate 619
## 16 ResultCommentText 383
## 17 DetectionQuantitationLimitMeasure.MeasureValue 373
## 18 TADA.DetectionQuantitationLimitMeasure.MeasureValue 358
## 19 SubjectTaxonomicName 278
## 20 ActivityLocation.LongitudeMeasure 272
## 21 ActivityLocation.LatitudeMeasure 269
## 22 MonitoringLocationIdentifier 227
## 23 MonitoringLocationName 222
## 24 LongitudeMeasure 218
## 25 TADA.LongitudeMeasure 218
## 26 LatitudeMeasure 215
## 27 TADA.LatitudeMeasure 214
## 28 TADA.ComparableDataIdentifier 197
## 29 ActivityDepthHeightMeasure.MeasureValue 166
## 30 TADA.ActivityDepthHeightMeasure.MeasureValue 163
## 31 CharacteristicName 144
## 32 TADA.CharacteristicName 143
## 33 DataQuality.UpperConfidenceLimitValue 95
## 34 ResultAnalyticalMethod.MethodIdentifier 74
## 35 ResultAnalyticalMethod.MethodName 72
## 36 ActivityEndDate 68
## 37 DataQuality.LowerConfidenceLimitValue 64
## 38 ActivityBottomDepthHeightMeasure.MeasureValue 55
## 39 TADA.ActivityBottomDepthHeightMeasure.MeasureValue 54
## 40 ResultAnalyticalMethod.MethodDescriptionText 41
## 41 ResultMeasure.MeasureUnitCode 39
## 42 ProjectName 30
## 43 ProjectIdentifier 30
## 44 TADA.ResultMeasure.MeasureUnitCode 22
## 45 TADA.DetectionQuantitationLimitMeasure.MeasureUnitCode 22
## 46 ProjectDescriptionText 18
## 47 SampleCollectionMethod.MethodName 15
## 48 CountyCode 15
## 49 HUCEightDigitCode 15
## 50 SampleCollectionMethod.MethodIdentifier 14
## 51 MonitoringLocationDescriptionText 14
## 52 DetectionQuantitationLimitMeasure.MeasureUnitCode 13
## 53 ResultAnalyticalMethod.MethodIdentifierContext 13
## 54 SampleCollectionMethod.MethodDescriptionText 13
## 55 SampleCollectionEquipmentName 11
## 56 MethodSpeciationName 10
## 57 TADA.ResultMeasureValueDataTypes.Flag 9
## 58 MeasureQualifierCode 9
## 59 LaboratoryName 9
## 60 ResultSampleFractionText 7
## 61 TADA.ResultSampleFractionText 7
## 62 ResultDetectionConditionText 7
## 63 DetectionQuantitationLimitTypeName 7
## 64 SampleCollectionMethod.MethodIdentifierContext 7
## 65 ActivityTypeCode 6
## 66 OrganizationIdentifier 6
## 67 OrganizationFormalName 6
## 68 MonitoringLocationTypeName 5
## 69 ActivityStartTime.TimeZoneCode 4
## 70 TADA.ActivityDepthHeightMeasure.MeasureValueDataTypes.Flag 4
## 71 QAPPApprovalAgencyName 4
## 72 StateCode 4
## 73 timeZoneStart 4
## 74 ActivityMediaName 3
## 75 TADA.ActivityMediaName 3
## 76 ActivityMediaSubdivisionName 3
## 77 ResultValueTypeName 3
## 78 TADA.DetectionQuantitationLimitMeasure.MeasureValueDataTypes.Flag 3
## 79 ResultDepthHeightMeasure.MeasureUnitCode 3
## 80 ActivityRelativeDepthName 3
## 81 TADA.ActivityBottomDepthHeightMeasure.MeasureValueDataTypes.Flag 3
## 82 StatisticalBaseCode 3
## 83 QAPPApprovedIndicator 3
## 84 ActivityEndTime.TimeZoneCode 3
## 85 timeZoneEnd 3
## 86 TADA.WQXResultUnitConversion 2
## 87 TADA.ResultDepthHeightMeasure.MeasureValueDataTypes.Flag 2
## 88 TADA.ResultDepthHeightMeasure.MeasureUnitCode 2
## 89 ActivityDepthHeightMeasure.MeasureUnitCode 2
## 90 TADA.ActivityDepthHeightMeasure.MeasureUnitCode 2
## 91 ActivityTopDepthHeightMeasure.MeasureValue 2
## 92 TADA.ActivityTopDepthHeightMeasure.MeasureValue 2
## 93 TADA.ActivityTopDepthHeightMeasure.MeasureValueDataTypes.Flag 2
## 94 ActivityTopDepthHeightMeasure.MeasureUnitCode 2
## 95 TADA.ActivityTopDepthHeightMeasure.MeasureUnitCode 2
## 96 ActivityBottomDepthHeightMeasure.MeasureUnitCode 2
## 97 TADA.ActivityBottomDepthHeightMeasure.MeasureUnitCode 2
## 98 CountryCode 2
## 99 HorizontalCoordinateReferenceSystemDatumName 2
## 100 ActivityConductingOrganizationText 2
## 101 ResultStatusIdentifier 2
## 102 SourceMapScaleNumeric 2
## 103 HorizontalCollectionMethodName 2
## 104 ProviderName 1Question 3: How many unique ‘TADA.ActivityMediaName’ values exist in your data frame? Are there any media types that are not water?
TADA is currently designed to accommodate water data from the WQP. Let’s ensure that we remove all non-water data first.
# remove data with media type that is not water
removed <- dataset_0 %>%
dplyr::filter(!TADA.ActivityMediaName %in% c("WATER")) %>%
dplyr::mutate(TADA.RemovalReason = "Activity media is not water.")
# what other media types exist in data frame?
unique(removed$TADA.ActivityMediaName)## [1] "BIOLOGICAL" "AIR"
# clean data frame containing only water
dataset <- dataset_0 %>% dplyr::filter(TADA.ActivityMediaName %in% c("WATER"))
dimCheck(all_result_num, dataset, removed, checkName = "Activity Media")## [1] "Good to go. Zero results created or destroyed in Activity Media check."Two additional helper functions one can use at any step in the
process are TADA::TADA_FieldValuesTable() and
TADA::TADA_FieldValuesPie(). These functions create a
summary table and pie chart (respectively) of all the unique values in a
given column. Let’s give it a try on OrganizationFormalName, which is a
WQP column naming the organization that supplied the result.
TADA::TADA_FieldValuesPie(dataset, field = "OrganizationFormalName")
org_counts <- TADA::TADA_FieldValuesTable(dataset, field = "OrganizationFormalName")
org_counts## Value Count
## 1 Red Lake DNR 85740
## 2 Fond du Lac Band of Chippewa (MN) 21063
## 3 Sac and Fox Nation (Tribal) 9943
## 4 Pueblo Of Tesuque (Tribal) 6798
## 5 Chickasaw Nation Environmental Service (Tribal) 4946
## 6 Pueblo of Pojoaque 1101Question 4: When might a user choose to view a column’s unique values as a table rather than in a pie chart?
We can take a quick look at some of the TADA-created columns that
review result value types. Because TADA is intended to work with numeric
data, at this point, it would be good to remove those result values that
are NA without any detection limit info, or contain text or special
characters that cannot be converted to numeric. Note that TADA will fill
in missing values with detection limit values and units with the
TADA::TADA_IDCensoredData if the
ResultDetectionConditionText and DetectionQuantitationLimitType fields
are populated. See ?TADA::TADA_ConvertSpecialChars for more
details on result value types and handling and
?TADA::TADA_IDCensoredData for details on censored data
preparation.
First, we can run TADA::TADA_IDCensoredData to fill in
as many NA/missing values as possible. We can use
TADA::TADA_FieldValuesPie to view the censored data flags
identified in the data frame and their relative frequency.
TADA::TADA_IDCensoredData sorts result values into
detection limit categories (e.g. non-detect, over-detect) based on
populated values in the ResultDetectionConditionText and
DetectionQuantitationLimitTypeName columns.
You can find the reference tables used to make these decisions in
TADA::TADA_GetDetCondRef() and
TADA::TADA_GetDetLimitRef() functions. In some cases,
results are missing detection limit/condition info, or there is a
conflict in the detection limit and condition. The user may want to
remove problematic detection limit data before proceeding. We can also
filter for the “problem” data by TADA.CensoredData.Flag and review the
unique reasons for data removal.
dataset <- TADA::TADA_IDCensoredData(dataset)## [1] "TADA_IDCensoredData: There are 115 results in your data frame that are missing ResultDetectionConditionText. TADA requires BOTH ResultDetectionConditionText and DetectionQuantitationLimitTypeName fields to be populated in order to categorize censored data."
## [1] "TADA_IDCensoredData: DetectionQuantitationLimitTypeName column in data frame contains value(s) NA which is/are not represented in the DetectionQuantitationLimitTypeName WQX domain table. These data records are placed under the TADA.CensoredData.Flag: Censored but not Categorized, and will not be used in censored data handling methods. Please contact TADA administrators to resolve."
TADA::TADA_FieldValuesPie(dataset, field = "TADA.CensoredData.Flag")
problem_censored <- dataset %>%
dplyr::filter(!TADA.CensoredData.Flag %in% c("Non-Detect", "Over-Detect", "Other", "Uncensored")) %>%
dplyr::mutate(TADA.RemovalReason = "Detection limit information contains errors or missing information.")
# Let's take a look at the problematic data that we filtered out (if any)
check <- unique(problem_censored[, c("TADA.CharacteristicName", "ResultDetectionConditionText", "DetectionQuantitationLimitTypeName", "TADA.CensoredData.Flag")])
check## TADA.CharacteristicName ResultDetectionConditionText
## 1 PHEOPHYTIN A <NA>
## 10 CHLOROPHYLL A <NA>
## 12 ORTHOPHOSPHATE <NA>
## 14 NITRATE <NA>
## 15 NITRITE <NA>
## 23 ALKALINITY, TOTAL <NA>
## 27 CHROMIUM <NA>
## 28 COPPER <NA>
## 104 SULFATE <NA>
## DetectionQuantitationLimitTypeName
## 1 <NA>
## 10 <NA>
## 12 Method Detection Level
## 14 Method Detection Level
## 15 Method Detection Level
## 23 Method Detection Level
## 27 Method Detection Level
## 28 Method Detection Level
## 104 Method Detection Level
## TADA.CensoredData.Flag
## 1 Detection condition is missing and required for censored data ID.
## 10 Detection condition is missing and required for censored data ID.
## 12 Detection condition is missing and required for censored data ID.
## 14 Detection condition is missing and required for censored data ID.
## 15 Detection condition is missing and required for censored data ID.
## 23 Detection condition is missing and required for censored data ID.
## 27 Detection condition is missing and required for censored data ID.
## 28 Detection condition is missing and required for censored data ID.
## 104 Detection condition is missing and required for censored data ID.
dataset <- dataset %>% dplyr::filter(TADA.CensoredData.Flag %in% c("Non-Detect", "Over-Detect", "Other", "Uncensored"))
# Let's take a look at the removed data
removed <- plyr::rbind.fill(removed, problem_censored)
# dimension check
dimCheck(all_result_num, dataset, removed, checkName = "Censored Data")## [1] "Good to go. Zero results created or destroyed in Censored Data check."Next, we can take a look at the data types present and filter out any non-allowable types.
# take a look at datatypes
flag.datatypes <- TADA::TADA_FieldValuesTable(dataset, field = "TADA.ResultMeasureValueDataTypes.Flag")
# Numeric or numeric-coerced data types
rv_datatypes <- unique(subset(dataset, !is.na(dataset$TADA.ResultMeasureValue))$TADA.ResultMeasureValueDataTypes.Flag)
# Non-numeric data types coerced to NA
na_rv_datatypes <- unique(subset(dataset, is.na(dataset$TADA.ResultMeasureValue))$TADA.ResultMeasureValueDataTypes.Flag)
# these are all of the NOT allowable data types in the dataset.
incompatible_datatype <- dataset %>%
dplyr::filter(!dataset$TADA.ResultMeasureValueDataTypes.Flag %in% c("Numeric", "Less Than", "Greater Than", "Approximate Value", "Percentage", "Comma-Separated Numeric", "Numeric Range - Averaged", "Result Value/Unit Copied from Detection Limit")) %>%
dplyr::mutate(TADA.RemovalReason = "Result value type cannot be converted to numeric or no detection limit values provided.")
# take a look at the difficult data types - do they make sense?
check <- unique(incompatible_datatype[, c("TADA.CharacteristicName", "ResultMeasureValue", "TADA.ResultMeasureValue", "ResultMeasure.MeasureUnitCode", "TADA.ResultMeasure.MeasureUnitCode", "TADA.ResultMeasureValueDataTypes.Flag", "DetectionQuantitationLimitMeasure.MeasureValue", "TADA.DetectionQuantitationLimitMeasure.MeasureValue", "DetectionQuantitationLimitMeasure.MeasureUnitCode", "TADA.DetectionQuantitationLimitMeasure.MeasureUnitCode")])
check## TADA.CharacteristicName
## 1 LOONS, VISUAL OBSERVATION
## 3 CLOUD COVER
## 6 WIND VELOCITY
## 7 CLOUD COVER
## 17 CLOUD COVER
## 23 HEIGHT, GAGE
## 32 TRUE COLOR
## 33 TOTAL HARDNESS
## 34 APPARENT COLOR
## 35 ALKALINITY
## 36 NITRATE + NITRITE
## 37 NITRATE + NITRITE
## 38 NITRATE + NITRITE
## 39 WATER APPEARANCE (TEXT)
## 40 WATER APPEARANCE (TEXT)
## 42 WATER APPEARANCE (TEXT)
## 44 WATER APPEARANCE (TEXT)
## 45 WATER APPEARANCE (TEXT)
## 47 WATER APPEARANCE (TEXT)
## 53 WATER APPEARANCE (TEXT)
## 69 WATER APPEARANCE (TEXT)
## 73 WATER APPEARANCE (TEXT)
## 90 WATER APPEARANCE (TEXT)
## 95 WATER APPEARANCE (TEXT)
## 98 WATER APPEARANCE (TEXT)
## 135 WATER APPEARANCE (TEXT)
## 167 WATER APPEARANCE (TEXT)
## 169 WATER APPEARANCE (TEXT)
## 219 WATER APPEARANCE (TEXT)
## 237 WATER APPEARANCE (TEXT)
## 283 WATER APPEARANCE (TEXT)
## 297 WATER APPEARANCE (TEXT)
## 372 WATER APPEARANCE (TEXT)
## 429 WATER APPEARANCE (TEXT)
## 495 WATER APPEARANCE (TEXT)
## 505 WATER APPEARANCE (TEXT)
## 511 WATER APPEARANCE (TEXT)
## 581 WATER APPEARANCE (TEXT)
## 601 WATER APPEARANCE (TEXT)
## 603 WATER APPEARANCE (TEXT)
## 675 WATER APPEARANCE (TEXT)
## 807 WATER APPEARANCE (TEXT)
## 820 WATER APPEARANCE (TEXT)
## 967 WATER APPEARANCE (TEXT)
## 1106 FLOW
## 1107 BAROMETRIC PRESSURE
## 1108 BAROMETRIC PRESSURE
## 1116 WATER APPEARANCE (TEXT)
## 1316 WATER APPEARANCE (TEXT)
## 1506 WATER APPEARANCE (TEXT)
## 1618 TEMPERATURE, WATER
## 1619 DISSOLVED OXYGEN SATURATION
## 1620 CONDUCTIVITY
## 1621 TOTAL DISSOLVED SOLIDS
## 1646 WATER APPEARANCE (TEXT)
## 1922 WATER APPEARANCE (TEXT)
## 2030 WATER APPEARANCE (TEXT)
## 2049 WATER APPEARANCE (TEXT)
## 2251 WATER APPEARANCE (TEXT)
## 2336 LAKE RECREATIONAL SUITABILITY (CHOICE LIST)
## 2376 LAKE RECREATIONAL SUITABILITY (CHOICE LIST)
## 2384 LAKE RECREATIONAL SUITABILITY (CHOICE LIST)
## 2430 WATER APPEARANCE (TEXT)
## ResultMeasureValue
## 1 <NA>
## 3 foggy
## 6 Calm
## 7 clear_sunny
## 17 raining
## 23 1:78
## 32 <NA>
## 33 <NA>
## 34 <NA>
## 35 <NA>
## 36 <NA>
## 37 <NA>
## 38 <NA>
## 39 Sediment; some algae present
## 40 Stain; some algae present
## 42 Sediment
## 44 Stain; Some algae present
## 45 Clear; Crystal Clear
## 47 Stain
## 53 Green; some algae present
## 69 Green; Some algae present
## 73 Stained; Crystal Clear
## 90 Green; crystal clear
## 95 Sediment; Some algae present
## 98 Clear water color; crystal clear physical conditions
## 135 Clear; some algae present
## 167 Stain; crystal clear
## 169 Green; definite algae present
## 219 Clear
## 237 Green
## 283 Tea-colored
## 297 Clear; Some algae present
## 372 Stain; Crystal clear
## 429 Cloudy
## 495 Water color clear; physical condition crystal clear
## 505 Water color stained; physical condition crystal clear
## 511 Stain; Definite algae present
## 581 Green; Crystal Clear
## 601 Sediment; Crystal clear
## 603 Sediment; Crystal Clear
## 675 Green; Definite algae present
## 807 Sediment; Definitie algae present
## 820 Green; Crystal clear
## 967 Stained
## 1106 <NA>
## 1107 600.7061 mmHG
## 1108 600.71 mmHG
## 1116 Clear; Crystal clear
## 1316 Sediment; Definite algae present
## 1506 Stain; definite algae present
## 1618 None
## 1619 mg/l
## 1620 mg/l
## 1621 PSS
## 1646 Clear; crystal clear
## 1922 Clear; Definite algae present
## 2030 Sediment; Stage tape-down from bridge
## 2049 Sediment; definite algae present
## 2251 Muddy
## 2336 1.VERY GOOD
## 2376 2.GOOD
## 2384 3.FAIR
## 2430 Green; High algal color
## TADA.ResultMeasureValue ResultMeasure.MeasureUnitCode
## 1 NA <NA>
## 3 NA %
## 6 NA mph
## 7 NA %
## 17 NA %
## 23 NA ft
## 32 NA CU
## 33 NA mg/L
## 34 NA CU
## 35 NA mg/L
## 36 NA mg/kg
## 37 NA mg/kg
## 38 NA mg/kg
## 39 NA <NA>
## 40 NA <NA>
## 42 NA <NA>
## 44 NA <NA>
## 45 NA <NA>
## 47 NA <NA>
## 53 NA <NA>
## 69 NA <NA>
## 73 NA <NA>
## 90 NA <NA>
## 95 NA <NA>
## 98 NA <NA>
## 135 NA <NA>
## 167 NA <NA>
## 169 NA <NA>
## 219 NA <NA>
## 237 NA <NA>
## 283 NA <NA>
## 297 NA <NA>
## 372 NA <NA>
## 429 NA <NA>
## 495 NA <NA>
## 505 NA <NA>
## 511 NA <NA>
## 581 NA <NA>
## 601 NA <NA>
## 603 NA <NA>
## 675 NA <NA>
## 807 NA <NA>
## 820 NA <NA>
## 967 NA <NA>
## 1106 NA cfs
## 1107 NA psi
## 1108 NA psi
## 1116 NA <NA>
## 1316 NA <NA>
## 1506 NA <NA>
## 1618 NA deg C
## 1619 NA %
## 1620 NA uS/cm
## 1621 NA mg/L
## 1646 NA <NA>
## 1922 NA <NA>
## 2030 NA <NA>
## 2049 NA <NA>
## 2251 NA <NA>
## 2336 NA <NA>
## 2376 NA <NA>
## 2384 NA <NA>
## 2430 NA <NA>
## TADA.ResultMeasure.MeasureUnitCode TADA.ResultMeasureValueDataTypes.Flag
## 1 <NA> NA - Not Available
## 3 % Text
## 6 M/SEC Text
## 7 % Text
## 17 % Text
## 23 IN Coerced to NA
## 32 PCU NA - Not Available
## 33 UG/L NA - Not Available
## 34 PCU NA - Not Available
## 35 UG/L NA - Not Available
## 36 MG/L NA - Not Available
## 37 MG/L NA - Not Available
## 38 MG/L NA - Not Available
## 39 <NA> Text
## 40 <NA> Text
## 42 <NA> Text
## 44 <NA> Text
## 45 <NA> Text
## 47 <NA> Text
## 53 <NA> Text
## 69 <NA> Text
## 73 <NA> Text
## 90 <NA> Text
## 95 <NA> Text
## 98 <NA> Text
## 135 <NA> Text
## 167 <NA> Text
## 169 <NA> Text
## 219 <NA> Text
## 237 <NA> Text
## 283 <NA> Text
## 297 <NA> Text
## 372 <NA> Text
## 429 <NA> Text
## 495 <NA> Text
## 505 <NA> Text
## 511 <NA> Text
## 581 <NA> Text
## 601 <NA> Text
## 603 <NA> Text
## 675 <NA> Text
## 807 <NA> Text
## 820 <NA> Text
## 967 <NA> Text
## 1106 CFS NA - Not Available
## 1107 G/M2 Text
## 1108 G/M2 Text
## 1116 <NA> Text
## 1316 <NA> Text
## 1506 <NA> Text
## 1618 DEG C Text
## 1619 % Text
## 1620 US/CM Text
## 1621 UG/L Text
## 1646 <NA> Text
## 1922 <NA> Text
## 2030 <NA> Text
## 2049 <NA> Text
## 2251 <NA> Text
## 2336 <NA> Text
## 2376 <NA> Text
## 2384 <NA> Text
## 2430 <NA> Text
## DetectionQuantitationLimitMeasure.MeasureValue
## 1 <NA>
## 3 <NA>
## 6 <NA>
## 7 <NA>
## 17 <NA>
## 23 <NA>
## 32 <NA>
## 33 <NA>
## 34 <NA>
## 35 <NA>
## 36 2.7
## 37 1.9
## 38 1.5
## 39 <NA>
## 40 <NA>
## 42 <NA>
## 44 <NA>
## 45 <NA>
## 47 <NA>
## 53 <NA>
## 69 <NA>
## 73 <NA>
## 90 <NA>
## 95 <NA>
## 98 <NA>
## 135 <NA>
## 167 <NA>
## 169 <NA>
## 219 <NA>
## 237 <NA>
## 283 <NA>
## 297 <NA>
## 372 <NA>
## 429 <NA>
## 495 <NA>
## 505 <NA>
## 511 <NA>
## 581 <NA>
## 601 <NA>
## 603 <NA>
## 675 <NA>
## 807 <NA>
## 820 <NA>
## 967 <NA>
## 1106 <NA>
## 1107 <NA>
## 1108 <NA>
## 1116 <NA>
## 1316 <NA>
## 1506 <NA>
## 1618 <NA>
## 1619 <NA>
## 1620 <NA>
## 1621 <NA>
## 1646 <NA>
## 1922 <NA>
## 2030 <NA>
## 2049 <NA>
## 2251 <NA>
## 2336 <NA>
## 2376 <NA>
## 2384 <NA>
## 2430 <NA>
## TADA.DetectionQuantitationLimitMeasure.MeasureValue
## 1 NA
## 3 NA
## 6 NA
## 7 NA
## 17 NA
## 23 NA
## 32 NA
## 33 NA
## 34 NA
## 35 NA
## 36 NA
## 37 NA
## 38 NA
## 39 NA
## 40 NA
## 42 NA
## 44 NA
## 45 NA
## 47 NA
## 53 NA
## 69 NA
## 73 NA
## 90 NA
## 95 NA
## 98 NA
## 135 NA
## 167 NA
## 169 NA
## 219 NA
## 237 NA
## 283 NA
## 297 NA
## 372 NA
## 429 NA
## 495 NA
## 505 NA
## 511 NA
## 581 NA
## 601 NA
## 603 NA
## 675 NA
## 807 NA
## 820 NA
## 967 NA
## 1106 NA
## 1107 NA
## 1108 NA
## 1116 NA
## 1316 NA
## 1506 NA
## 1618 NA
## 1619 NA
## 1620 NA
## 1621 NA
## 1646 NA
## 1922 NA
## 2030 NA
## 2049 NA
## 2251 NA
## 2336 NA
## 2376 NA
## 2384 NA
## 2430 NA
## DetectionQuantitationLimitMeasure.MeasureUnitCode
## 1 <NA>
## 3 <NA>
## 6 <NA>
## 7 <NA>
## 17 <NA>
## 23 <NA>
## 32 <NA>
## 33 <NA>
## 34 <NA>
## 35 <NA>
## 36 mg/kg
## 37 mg/kg
## 38 mg/kg
## 39 <NA>
## 40 <NA>
## 42 <NA>
## 44 <NA>
## 45 <NA>
## 47 <NA>
## 53 <NA>
## 69 <NA>
## 73 <NA>
## 90 <NA>
## 95 <NA>
## 98 <NA>
## 135 <NA>
## 167 <NA>
## 169 <NA>
## 219 <NA>
## 237 <NA>
## 283 <NA>
## 297 <NA>
## 372 <NA>
## 429 <NA>
## 495 <NA>
## 505 <NA>
## 511 <NA>
## 581 <NA>
## 601 <NA>
## 603 <NA>
## 675 <NA>
## 807 <NA>
## 820 <NA>
## 967 <NA>
## 1106 <NA>
## 1107 <NA>
## 1108 <NA>
## 1116 <NA>
## 1316 <NA>
## 1506 <NA>
## 1618 <NA>
## 1619 <NA>
## 1620 <NA>
## 1621 <NA>
## 1646 <NA>
## 1922 <NA>
## 2030 <NA>
## 2049 <NA>
## 2251 <NA>
## 2336 <NA>
## 2376 <NA>
## 2384 <NA>
## 2430 <NA>
## TADA.DetectionQuantitationLimitMeasure.MeasureUnitCode
## 1 <NA>
## 3 %
## 6 M/SEC
## 7 %
## 17 %
## 23 IN
## 32 PCU
## 33 UG/L
## 34 PCU
## 35 UG/L
## 36 MG/L
## 37 MG/L
## 38 MG/L
## 39 <NA>
## 40 <NA>
## 42 <NA>
## 44 <NA>
## 45 <NA>
## 47 <NA>
## 53 <NA>
## 69 <NA>
## 73 <NA>
## 90 <NA>
## 95 <NA>
## 98 <NA>
## 135 <NA>
## 167 <NA>
## 169 <NA>
## 219 <NA>
## 237 <NA>
## 283 <NA>
## 297 <NA>
## 372 <NA>
## 429 <NA>
## 495 <NA>
## 505 <NA>
## 511 <NA>
## 581 <NA>
## 601 <NA>
## 603 <NA>
## 675 <NA>
## 807 <NA>
## 820 <NA>
## 967 <NA>
## 1106 CFS
## 1107 G/M2
## 1108 G/M2
## 1116 <NA>
## 1316 <NA>
## 1506 <NA>
## 1618 DEG C
## 1619 %
## 1620 US/CM
## 1621 UG/L
## 1646 <NA>
## 1922 <NA>
## 2030 <NA>
## 2049 <NA>
## 2251 <NA>
## 2336 <NA>
## 2376 <NA>
## 2384 <NA>
## 2430 <NA>Then we can take a closer look at the removed results and run another dimension check on the data set.
# filter data set to include allowable data types
dataset <- dataset %>% dplyr::filter(dataset$TADA.ResultMeasureValueDataTypes.Flag %in% c("Numeric", "Less Than", "Greater Than", "Approximate Value", "Percentage", "Comma-Separated Numeric", "Numeric Range - Averaged", "Result Value/Unit Copied from Detection Limit"))
# create data frame to includ all removed results
removed <- plyr::rbind.fill(removed, incompatible_datatype)
# dimension check
dimCheck(all_result_num, dataset, removed, checkName = "Result Format")## [1] "Good to go. Zero results created or destroyed in Result Format check."Data flagging
We’ve taken a quick look at the raw data frame and split off some
data that are not compatible with TADA, now let’s run through some
quality control checks. The most important ones to run to ensure your
data frame is ready for subsequent steps are
TADA::TADA_FlagFraction(),
TADA::TADA_FlagSpeciation(),
TADA::TADA_FlagResultUnit(), and
TADA::TADA_FindQCActivities(). With the exception of
TADA::TADA_FindQCActivities(), these flagging functions
leverage WQX’s QAQC
Validation Table. See the WQX
QAQC Service User Guide for more details on how TADA leverages the
validation table to flag potentially invalid data.
TADA::TADA_FindQCActivities() uses a TADA-specific domain
table users can review with
TADA::TADA_GetActivityTypeRef(). All QAQC tables are
frequently updated in the package to ensure they match the latest
version on the web.
Bring the QAQC Validation Table into your R session to view or save with the following command:
qaqc_ref <- TADA::TADA_GetWQXCharValRef()
unique(qaqc_ref$Type)## [1] "CharacteristicFraction" "CharacteristicMethod"
## [3] "CharacteristicSpeciation" "CharacteristicUnit"Question 5: What do you think the qaqc_ref$Type
column indicates?
TADA joins this validation table to the data and uses the “Valid” and “Invalid” labels in the Status column to create easily understandable flagging columns for each function. Let’s run these four flagging functions.
dataset_flags <- TADA::TADA_FlagFraction(dataset, clean = FALSE, flaggedonly = FALSE)## [1] "Rows with invalid sample fractions have been flagged but retained. Review these rows before proceeding and/or set clean = TRUE."
dataset_flags <- TADA::TADA_FlagSpeciation(dataset_flags, clean = "none", flaggedonly = FALSE)## [1] "Rows with invalid speciations have been flagged but retained. Review these rows before proceeding and/or set clean = 'invalid_only' or 'both'."
dataset_flags <- TADA::TADA_FlagResultUnit(dataset_flags, clean = "none", flaggedonly = FALSE)## [1] "Rows with invalid result value units have been flagged but retained. Review these rows before proceeding and/or set clean = 'invalid_only' or 'both'."
dataset_flags <- TADA::TADA_FindQCActivities(dataset_flags, clean = FALSE, flaggedonly = FALSE)
dimCheck(all_result_num, dataset_flags, removed, checkName = "Run Flag Functions")## [1] "Good to go. Zero results created or destroyed in Run Flag Functions check."Question 6: Did any warnings or messages appear in the console after running these flagging functions? What do they say?
Now that we’ve run all the key flagging functions, let’s take a look at the results and make some decisions.
TADA::TADA_FieldValuesPie(dataset_flags, field = "TADA.SampleFraction.Flag")
TADA::TADA_FieldValuesPie(dataset_flags, field = "TADA.MethodSpeciation.Flag")
TADA::TADA_FieldValuesPie(dataset_flags, field = "TADA.ResultUnit.Flag")
TADA::TADA_FieldValuesPie(dataset_flags, field = "TADA.ActivityType.Flag")
Any results flagged as “Invalid” are recognized in the QAQC Validation Table as having some data quality issue. “NonStandardized” means that the format has not been fully vetted or processed, while “Valid” confirms that the characteristic combination is widely recognized as correctly formatted. Let’s add any invalid results to the removed data frame for later review.
Note: if you find any errors in the QAQC Validation Table, please contact the WQX Help Desk at WQX@epa.gov to help correct it. Thanks in advance!
# grab all the flagged results from the four functions
problem_flagged <- dataset_flags %>%
filter(TADA.SampleFraction.Flag == "Invalid" | TADA.MethodSpeciation.Flag == "Invalid" | TADA.ResultUnit.Flag == "Invalid" | !TADA.ActivityType.Flag %in% ("Non_QC")) %>%
dplyr::mutate(TADA.RemovalReason = "Invalid Unit, Method, Speciation, or Activity Type.")
dataset_flags <- dataset_flags %>% dplyr::filter(!ResultIdentifier %in% problem_flagged$ResultIdentifier)
# create data frame of removed results
removed <- plyr::rbind.fill(removed, problem_flagged)
# remove df no longer needed
rm(problem_flagged)
# dimension check
dimCheck(all_result_num, dataset_flags, removed, checkName = "Filter Flag Functions")## [1] "Good to go. Zero results created or destroyed in Filter Flag Functions check."Question 7: Are there any other metadata columns that you review and filter in your workflow?
We’ve finished running the recommended flagging functions and removing results that do not pass QC checks. Let’s look at the breakdown of these data in the removed object.
removal <- TADA::TADA_FieldValuesTable(removed, field = "TADA.RemovalReason")
removal## Value
## 1 Activity media is not water.
## 2 Result value type cannot be converted to numeric or no detection limit values provided.
## 3 Invalid Unit, Method, Speciation, or Activity Type.
## 4 Detection limit information contains errors or missing information.
## Count
## 1 4459
## 2 3187
## 3 1421
## 4 115You can review any other columns of interest and create custom domain tables of your “Valid” and “Invalid” criteria using R or Excel. Also check out some of the other flagging functions available in TADA:
?TADA_FindNearbySites()- under active development?TADA_FindPotentialDuplicatesMultipleOrgs()?TADA_FindPotentialDuplicatesSingleOrg()?TADA_FindQAPPApproval()?TADA_FindQAPPDoc()?TADA_FlagAboveThreshold()?TADA_FlagBelowThreshold()?TADA_FlagContinuousData()?TADA_FlagCoordinates()?TADA_FlagMeasureQualifierCode()?TADA_FlagMethod()
Please let us know of other flagging functions you think would have broad appeal in the TADA package or need assistance brainstorming/developing.
Censored data handling
We have already identified, flagged, and in some cases removed problematic detection limit data from our data frame, but that doesn’t keep them from being difficult. Because we do not know the result value with adequate precision, water quality data users often set non-detect values to some number below the reported detection limit. TADA contains some simple methods for handling detection limits: users may multiply the detection limit by some number between 0 and 1, or convert the detection limit value to a random number between 0 and the detection limit. More complex detection limit estimation requiring regression models (Maximum Likelihood, Kaplan-Meier, Robust Regression on Order Statistics) or similar must be performed outside of the current version of TADA (though future development is planned).
Question 8: How would you parameterize
TADA_SimpleCensoredMethods() to make non-detect values
equal to the provided detection limit? What would you need to change in
the example below?
dataset_cens <- TADA::TADA_SimpleCensoredMethods(dataset_flags, nd_method = "multiplier", nd_multiplier = 0.5, od_method = "as-is")Let’s take a look at how the censored data handling function affects
the TADA.ResultMeasureValueDataTypes.Flag column.
First, we can look use TADA_FieldValuesTable to look at
the TADA.ResultMeasureValueDataTypes.Flag column in data set before we
ran TADA_SimpleCensoredMethods.
# before
TADA::TADA_FieldValuesTable(dataset_flags, field = "TADA.ResultMeasureValueDataTypes.Flag")## Value Count
## 1 Numeric 120031
## 2 Result Value/Unit Copied from Detection Limit 4029
## 3 Percentage 752
## 4 Numeric Range - Averaged 33
## 5 Less Than 19
## 6 Greater Than 3
## 7 Comma-Separated Numeric 1Then we can use TADA_FieldValuesTable again to look at
the same column after TADA_SimpleCensoredMethods.
# after
TADA::TADA_FieldValuesTable(dataset_cens, field = "TADA.ResultMeasureValueDataTypes.Flag")## Value Count
## 1 Numeric 120012
## 2 Result Value/Unit Estimated from Detection Limit 4043
## 3 Percentage 752
## 4 Numeric Range - Averaged 33
## 5 Less Than 19
## 6 Result Value/Unit Copied from Detection Limit 5
## 7 Greater Than 3
## 8 Comma-Separated Numeric 1Question 9: Is there a difference between the first and second tables?
If you’d like to start thinking about using statistical methods to
estimate detection limit values, check out the ?TADA_Stats
function, which accepts user-defined data groupings (or defaults to
TADA.ComparableDataIdentifier to determine measurement count, location
count, censored data stats, min, max, and percentile stats, and suggests
non-detect estimatiom methods based on the number of results, % of data
frame censored, and number of censoring levels (detection limits). The
decision tree used in the function was outlined in a National
Nonpoint Source Tech Memo.
Data exploration
How are you feeling about your test data frame? Does it seem ready
for the next step(s) in your analyses? There’s probably a lot more you’d
like to look at/filter out before you’re ready to say: QC complete.
Let’s first check out characteristics in the data frame using
dplyr functions and pipes.
# get table of characteristics with number of results, sites, and organizations
dataset_cens_summary <- dataset_cens %>%
dplyr::group_by(TADA.CharacteristicName) %>%
dplyr::summarise(Result_Count = length(ResultIdentifier), Site_Count = length(unique(MonitoringLocationIdentifier)), Org_Count = length(unique(OrganizationIdentifier))) %>%
dplyr::arrange(desc(Result_Count)) You may see a characteristic that you’d like to investigate further
in isolation. TADA_FieldValuesPie() will also produce
summary pie charts for a given column within a specific
characteristic. Let’s take a look.
# go ahead and pick a characteristic name from the table generated above. I picked dissolved oxygen (DO) amd selected OrganizationFormalName as the field to see the relative contribution of each org to DO results
TADA::TADA_FieldValuesPie(dataset_cens, field = "OrganizationFormalName", characteristicName = "DISSOLVED OXYGEN (DO)")
We can view the site locations using a TADA mapping function. In this map, the circles indicate monitoring locations in the data set; their size corresponds to the number of results collected at that site, while the darker the circle, the more characteristics were sampled at that site.
TADA::TADA_OverviewMap(dataset_cens)Out of curiosity, let’s take a look at a breakdown of these monitoring location types. Do they all indicate surface water samples? Depending upon your program’s goals and methods, you might want to filter out some of the types you see.
TADA::TADA_FieldValuesPie(dataset_cens, field = "MonitoringLocationTypeName")
One of the next big steps is data harmonization: translating and
aggregating synonyms, combining multiple forms/species of certain
characteristics, etc. We won’t get to that in this demo (more details
can be found here: TADA
Module 1: Water Quality Portal Data Discovery and Cleaning or TADA_HarmonizeSynonyms()),
but for now we can start looking at data distributions within a single
characteristic-speciation-fraction-unit using the plotting functions
TADA_Histogram() and TADA_Boxplot(). We can
also view a stats table using TADA_Stats.
Let’s first take a look at the column TADA.ComparableDataIdentifier, which breaks down characteristics into groups by name, fraction, speciation, and unit. These four columns are important to evaluate together to ensure the plotted data are sufficiently similar to appear on a single plot together: it doesn’t make sense to plot characteristics with different units or fractions in the same distribution.
# trusty field values table - lets just look at the first few entries with the most associated records
compid <- TADA::TADA_FieldValuesTable(dataset_cens, field = "TADA.ComparableDataIdentifier")Now that we have an idea for what the TADA.ComparableDataIdentifier looks like, we can check out how it is used to plot distinct characteristic groups.
# Look at a histogram, boxplot, and stats for TADA.ComparableDataIdentifier(s) of your choice.
comp_data_id <- "PH_NA_NA_NONE"
plot_data <- subset(dataset_cens, dataset_cens$TADA.ComparableDataIdentifier %in% comp_data_id)Question 10: How does selecting the different options on the left side of the histogram change the data displayed? When might you want to use a histogram vs. a boxplot?
Let’s take a look at the histogram and boxplot for the comparable data identifier we selected.
TADA::TADA_Histogram(plot_data, id_cols = "TADA.ComparableDataIdentifier")## [1] "Plotting function removed 7 results where TADA.ResultMeasureValue = NA. These results cannot be plotted."
TADA::TADA_Boxplot(plot_data, id_cols = "TADA.ComparableDataIdentifier")## [1] "Plotting function removed 7 results where TADA.ResultMeasureValue = NA. These results cannot be plotted."
stats <- TADA::TADA_Stats(plot_data)## [1] "Dataset contains 7 results missing both a TADA result value and a detection limit. These values will not be represented in the stats summary table. Suggest removing or handling."We can also explore depth profiles for selected characteristics at
specific site on a single date. There are a few functions that can help
with this. First we can use TADA_FlagDepthCategory to place
results into various depth categories (surface, middle, and bottom).
dataset_depth <- TADA::TADA_FlagDepthCategory(dataset_cens)## [1] "TADA_FlagDepthCategory: checking data set for depth values. 57671 results have depth values available."
## [1] "TADA_FlagDepthCategory: assigning depth categories."
## [1] "TADA_FlagDepthCategory: Grouping results by MonitoringLocationIdentifier, OrganizationIdentifier, CharacteristicName, and ActivityStartDate for aggregation for entire water column."
## [1] "TADA_FlagDepthCategory: No aggregation performed."We can also use another function, TADA_IDDepthProfiles
to identify location/date/characteristic combinations in the data set
that can be used for depth profile plots or analysis. The default number
of values required to identify a location/date/characteristic as a depth
profile is 2, but this can be changed by the user. We will specify a
larger value, 5, so that any depth profiles identified will have results
from at least 5 different depths.
depth_profile_id <- TADA::TADA_IDDepthProfiles(dataset_depth, nvalue = 5)Question 11: How can TADA_IDDepthProfiles() help users use TADA_DepthProfilePlot most efficiently?
Now, we can use TADA_DepthProfilePlot to plot up to
three characteristics against depth. In this example, we will look at
pH, secchi depth, and pH.
TADA::TADA_DepthProfilePlot(dataset_cens,
groups = c('TEMPERATURE,
WATER_NA_NA_DEG C',
'DEPTH, SECCHI DISK DEPTH_NA_NA_M',
'PH_NA_NA_NONE'),
location = "REDLAKE_WQX-ANKE",
activity_date = "2018-10-04",
depthcat = TRUE,
surfacevalue = 2,
bottomvalue = 2,
unit = "m")## [1] "TADA_DepthProfilePlot: Running TADA_DepthCategoryFlag function to add required columns to data frame"
## [1] "TADA_FlagDepthCategory: checking data set for depth values. 57671 results have depth values available."
## [1] "TADA_FlagDepthCategory: assigning depth categories."
## [1] "TADA_FlagDepthCategory: Grouping results by MonitoringLocationIdentifier, OrganizationIdentifier, CharacteristicName, and ActivityStartDate for aggregation for entire water column."
## [1] "TADA_FlagDepthCategory: No aggregation performed."
## [1] "TADA_DepthProfilePlot: Depth unit in data set matches depth unit specified by user for plot. No conversion necessary."
## [1] "TADA_DepthProfilePlot: Identifying available depth profile data."
## [1] "TADA_DepthProfilePlot: Any results for DEPTH, SECCHI DISK DEPTH, DEPTH, SECCHI DISK DEPTH (CHOICE LIST), DEPTH, SECCHI DISK DEPTH REAPPEARS, DEPTH, DATA-LOGGER (NON-PORTED), DEPTH, DATA-LOGGER (PORTED), RBP STREAM DEPTH - RIFFLE, RBP STREAM DEPTH - RUN, THALWEG DEPTH match the depth unit selected for the figure."
## [1] "TADA_DepthProfilePlot: Adding surface delination to figure."
## [1] "TADA_DepthProfilePlot: Adding bottom delination to figure."Finally, we can download our PASS and FAIL data sets together into an Excel spreadsheet.
dataset_and_removed <- dplyr::bind_rows(dataset_cens, removed)
# Un-comment to download Excel spreadsheet to your working directory
# install.packages(writexl)
# library(writexl)
# writexl::write_xlsx(dataset_and_removed, "NCTCShepherdstownData.xlsx")TADA R Shiny Modules
Finally, take a look at an alternative workflow for QC’ing WQP data: TADA Shiny Module 1: Data Discovery and Cleaning. This is a Shiny application that runs many of the TADA functions covered in this training document behind a graphical user interface. The shiny application queries the WQP, contains maps and data visualizations, flags suspect data results, handles censored data, and more. You can launch it using the code below.
# download TADA Shiny repository
remotes::install_github("USEPA/TADAShiny", ref = "develop", dependencies = TRUE)
# launch the app locally.
TADAShiny::run_app()DRAFT Module 1 is also currently hosted on the web with minimal server memory/storage allocated.